CodePipeline を利用した ECS Service の自動リリースをやってみた
ECS を利用したアプリケーションを構築する上でデプロイ戦略の選択は重要な要素になります. 開発段階では ECS が制御するローリングアップデートを利用して手動デプロイすることはあっても, 本番環境では自動デプロイが好ましいでしょう. 今回はCodePipelineを主軸に自動デプロイできる環境を構築します.
デプロイ戦略の概要について
今回はパイプライン内でDocker Imageをビルド, pushを行った後に, ECS ServiceをBlue/Greenデプロイします.
またDocker Imageに付与するタグとしてgitのコミットハッシュを利用することでコードとイメージを一意に紐づけることも同様に行います.
パイプラインの構築は主にTerraformを利用しますが, 一部AWS CLIを利用します.
今回は検証が目的のため, 本番環境を意識したコード分割などは行っていません.
CodeBuildの環境構築
GitHubの特定リポジトリにpushされた際にDocker Imageをビルドして, ECRリポジトリにpushするようにCodeBuild プロジェクトを作成していきます. Docker Imageをビルドする際に, Git Commit HashをDocker Image Tagとして利用します. このことでDocker Imageを一意にすることが容易にできます.
まずはTerraformの初期設定とECS Cluster, ECR リポジトリのコーディングを行います.
terraform { required_version = ">= 0.12" } provider "aws" { region = "ap-northeast-1" }
次にECS Clusterをコーディングします.
resource "aws_ecs_cluster" "main" { name = "mycluster" capacity_providers = ["FARGATE", "FARGATE_SPOT"] default_capacity_provider_strategy { capacity_provider = "FARGATE_SPOT" weight = 1 } }
今回はDocker Imageにgo/serverという名前をつけるのでaws_ecr_reposityのname部分でgo/serverを指定しています.
resource "aws_ecr_repository" "main" { name = "go/server" image_tag_mutability = "MUTABLE" }
ここまでコードを書いたら一度applyを実行します.
$ terraform init $ terraform apply
次にCodePipelineをコーディングしていきます. IAM RoleやS3, CodeBuildプロジェクトなど, まだ書いていないリソースは後ほど記載します. Source ステージにある, configuration部分は利用するGitHubリポジトリとブランチを記載しましょう.
resource "aws_codepipeline" "main" { name = "ecs-pipeline" role_arn = aws_iam_role.codepipeline.arn artifact_store { type = "S3" location = aws_s3_bucket.pipeline_artifact.id } stage { name = "Source" action { name = "Source" category = "Source" owner = "ThirdParty" provider = "GitHub" version = "1" run_order = 1 output_artifacts = ["source"] configuration = { Owner = "YOUR GITHUB ACCOUNT" Repo = "YOUR GITHUB REPOSITORY" Branch = "YOUR GITHUB BRANCH" } } } stage { name = "Build" action { name = "Build" category = "Build" owner = "AWS" provider = "CodeBuild" version = "1" run_order = 2 input_artifacts = ["source"] output_artifacts = ["build"] configuration = { ProjectName = "ecs-pipeline" } } } }
CodePipelineでは各ステージでのアーティファクト管理をS3バケットで行います. なのでCodePipelineで利用するS3バケットのコーディングします.
resource "aws_s3_bucket" "pipeline_artifact" { acl = "private" }
CodeBuild プロジェクトのコーディングをしていきます. この際いくつか注意する事項があります.
- artifacts, source にCODEPIPELINEを指定すること
- environmentでprivileged_modeを有効にすること
1つ目ですが, CodePipelineを利用する場合はアーティファクトのやりとりをS3バケットで行います.
なのでartifacts, source にCODEPIPELINEを指定する必要があります.
2つ目ですが, 今回はDocker Imageをビルドする都合上, privileged_modeを有効にする必要があります.
以上の点を忘れて何回かビルドに失敗しているので気をつけましょう.
また今回はログの出力先であるCloudWatch Logs Groupを指定し, 後ほどコーディングします.
ここで指定をしないと, CodeBuildがCloudWatch Logs Groupを自動で作成するのでterraformでリソースを削除してもログが残ってしまいます.
resource "aws_codebuild_project" "main" { name = "ecs-pipeline" description = "ecs pipeline" build_timeout = 60 service_role = aws_iam_role.codebuild.arn artifacts { type = "CODEPIPELINE" } cache { type = "LOCAL" modes = ["LOCAL_CUSTOM_CACHE"] } logs_config { cloudwatch_logs { status = "ENABLED" group_name = aws_cloudwatch_log_group.codebuild.name } s3_logs { status = "DISABLED" } } source { type = "CODEPIPELINE" buildspec = "buildspec.yaml" } environment { compute_type = "BUILD_GENERAL1_SMALL" image = "aws/codebuild/standard:3.0" type = "LINUX_CONTAINER" image_pull_credentials_type = "CODEBUILD" privileged_mode = true } }
先ほどの記載通りCloudWatch Logs Groupを作成します.
resource "aws_cloudwatch_log_group" "codebuild" { name = "/codebuild/ecs-pipeline" }
最後にCodePipeline, CodeBuildで利用するIAM ロールを作成します.
長いので分割しますが, 環境に合わせてよしなに権限を絞り込んでください.
iam_role.tf(CodePipeline)
data "aws_iam_policy_document" "codepipeline_assumerole" { statement { actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = ["codepipeline.amazonaws.com"] } } } resource "aws_iam_role" "codepipeline" { name = "ecs-pipeline-project" assume_role_policy = data.aws_iam_policy_document.codepipeline_assumerole.json } resource "aws_iam_policy" "codepipeline" { name = "ecs-pipeline-codepipeline" description = "ecs-pipeline-codepipeline" policy = templatefile("${path.root}/assets/codepipeline_policy.tpl", { artifacts = aws_s3_bucket.pipeline_artifact.id }) } resource "aws_iam_role_policy_attachment" "codepipeline" { role = aws_iam_role.codepipeline.id policy_arn = aws_iam_policy.codepipeline.arn }
codepipeline_policy.tpl
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "iam:PassRole" ], "Resource": "*", "Effect": "Allow", "Condition": { "StringEqualsIfExists": { "iam:PassedToService": [ "cloudformation.amazonaws.com", "elasticbeanstalk.amazonaws.com", "ec2.amazonaws.com", "ecs-tasks.amazonaws.com" ] } } }, { "Sid": "CodeBuildPolicy", "Effect": "Allow", "Action": [ "codebuild:BatchGetBuilds", "codebuild:StartBuild" ], "Resource": "*" }, { "Sid": "S3Policy", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::${artifacts}", "arn:aws:s3:::${artifacts}/*" ] }, { "Action": [ "codedeploy:CreateDeployment", "codedeploy:GetApplication", "codedeploy:GetApplicationRevision", "codedeploy:GetDeployment", "codedeploy:GetDeploymentConfig", "codedeploy:RegisterApplicationRevision" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "elasticloadbalancing:*", "cloudwatch:*", "sns:*", "ecs:*" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "lambda:InvokeFunction", "lambda:ListFunctions" ], "Resource": "*", "Effect": "Allow" }, { "Effect": "Allow", "Action": [ "ecr:DescribeImages" ], "Resource": "*" } ] }
iam_role.tf (CodeBuild)
data "aws_iam_policy_document" "codebuild_assumerole" { statement { actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = ["codebuild.amazonaws.com"] } } } data "aws_iam_policy_document" "codebuild" { statement { effect = "Allow" actions = [ "ecr:GetAuthorizationToken" ] resources = [ "*" ] } statement { effect = "Allow" actions = [ "ecr:*" ] resources = [ aws_ecr_repository.main.arn ] } statement { effect = "Allow" actions = [ "s3:List*", "s3:Get*", "s3:PutObject" ] resources = [ aws_s3_bucket.pipeline_artifact.arn, "${aws_s3_bucket.pipeline_artifact.arn}/*" ] } statement { effect = "Allow" actions = [ "logs:CreateLogStream", "logs:PutLogEvents" ] resources = [ aws_cloudwatch_log_group.codebuild.arn, ] } } resource "aws_iam_role" "codebuild" { name = "ecs-pipeline-codebuild" assume_role_policy = data.aws_iam_policy_document.codebuild_assumerole.json } resource "aws_iam_role_policy" "codebuild" { name = "ecs-pipeline-codebuild" role = aws_iam_role.codebuild.id policy = data.aws_iam_policy_document.codebuild.json }
ここまででAWS側のCodeBuildの実行準備が完了しました. TerraformでGitHubをCodePipelineのSourceとする場合に, GitHubのPersonal Access Tokenが必要になります. こちら のガイドを参考に取得しましょう. リポジトリへの操作権限があれば十分です.
$ export GITHUB_TOKEN=YOUR GITHUB PERSONAL ACCESS TOKEN $ terraform apply
最後にリポジトリ側の準備を行い, 実際にビルドがうまくいくかを確認します. 今回は新たにGoを利用してWebサーバを作成します.
$ go mod init github.com/YOUR_REPORITORY
main.goを記載してHnet/http を利用し, Hello Worldとレスポンスするサーバを作成します.
package main import ( "fmt" "net/http" ) func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "Hello World\n") }) http.ListenAndServe(":80", nil) }
またこのコードを元にDocker Imageを作成できるようにDockerfileを記載します.
FROM golang:latest RUN mkdir /app COPY . /app WORKDIR /app RUN go build -o main main.go ENTRYPOINT ["/app/main"] EXPOSE 80
最後にbuildspecを記載して, Docker Imageの作成とECRリポジトリへのpushをCodeBuildが自動で行ってくれるようにします.
CodePipelineを利用する場合はSourceがS3バケットになるためGit Commit Hashの取得に$CODEBUILD_RESOLVED_SOURCE_VERSION
を利用する必要があります.
またimageDetail.jsonというアーティファクトを出力していますが, これはCodeDeployでECS Serviceをデプロイする際に利用するものです.
version: 0.2 env: variables: IMAGE_NAME: 'go/server' phases: install: runtime-versions: docker: 18 pre_build: commands: - $(aws ecr get-login --no-include-email --region ${AWS_DEFAULT_REGION}) - AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text) - URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_NAME} build: commands: - docker build -t $URI:$CODEBUILD_RESOLVED_SOURCE_VERSION . - docker push $URI:$CODEBUILD_RESOLVED_SOURCE_VERSION - printf '{"Version":"1.0","ImageURI":"%s"}' $URI:$CODEBUILD_RESOLVED_SOURCE_VERSION > imageDetail.json artifacts: files: imageDetail.json
ここまで終了したらコードをリポジトリにpushしてパイプラインの動作とECR リポジトリを確認しましょう. 問題なければCodeDeployの設定に移ります.
CodeDeployの環境構築
本来であれば, VPCやALBなどは既存のものがあり別途管理してると思いますが, 今回は検証が目的のためVPCとALBがまだありません. なのでCodeDeployの設定時に必要になるためこのタイミングで作成します. まずは, VPCとALB, ECS Serviceで必要になるSecurity Groupのコーディングをします.
locals { vpc_cidr = "192.168.0.0/16" subnet_numbers = { "ap-northeast-1a" = 0 "ap-northeast-1c" = 1 } } resource "aws_vpc" "main" { cidr_block = local.vpc_cidr enable_dns_hostnames = "true" enable_dns_support = "true" } resource "aws_subnet" "public" { for_each = local.subnet_numbers vpc_id = aws_vpc.main.id cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, each.value) availability_zone = each.key } resource "aws_internet_gateway" "main" { vpc_id = aws_vpc.main.id } resource "aws_route_table" "public" { vpc_id = aws_vpc.main.id route { cidr_block = "0.0.0.0/0" gateway_id = aws_internet_gateway.main.id } } resource "aws_route_table_association" "public" { for_each = local.subnet_numbers route_table_id = aws_route_table.public.id subnet_id = aws_subnet.public[each.key].id } resource "aws_security_group" "lb" { vpc_id = aws_vpc.main.id name = "go-server-alb" description = "go-server-alb" ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = [ "0.0.0.0/0" ] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } resource "aws_security_group" "service" { vpc_id = aws_vpc.main.id name = "go-server-service" description = "go-server-service" ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = [ "0.0.0.0/0" ] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } }
次にALBのコーディングを行います. CodeDeployを利用して, ECSにBlue/Green デプロイメントを行う場合, ターゲットグループが2つ必要になります. 忘れずに作成しましょう.
resource "aws_lb" "main" { name = "go-server-alb" internal = false load_balancer_type = "application" enable_deletion_protection = false security_groups = [ aws_security_group.lb.id ] subnets = [ aws_subnet.public["ap-northeast-1a"].id, aws_subnet.public["ap-northeast-1c"].id ] } resource "aws_lb_target_group" "blue" { name = "blue" port = 80 protocol = "HTTP" vpc_id = aws_vpc.main.id target_type = "ip" health_check { protocol = "HTTP" path = "/" interval = 30 timeout = 5 healthy_threshold = 3 unhealthy_threshold = 2 matcher = 200 } } resource "aws_lb_target_group" "green" { name = "green" port = 80 protocol = "HTTP" vpc_id = aws_vpc.main.id target_type = "ip" health_check { protocol = "HTTP" path = "/" interval = 30 timeout = 5 healthy_threshold = 3 unhealthy_threshold = 2 matcher = 200 } } resource "aws_lb_listener" "main" { load_balancer_arn = aws_lb.main.arn port = "80" protocol = "HTTP" default_action { type = "forward" target_group_arn = aws_lb_target_group.green.arn } }
最後にECS Serviceの作成などでALBのーターゲットグループARNやセキュリティグループのIDが必要になるので出力させます.
output "public_subnet1" { value = aws_subnet.public["ap-northeast-1a"].id } output "public_subnet2" { value = aws_subnet.public["ap-northeast-1c"].id } output "service_sg" { value = aws_security_group.service.id } output "blue_tg_arn" { value = aws_lb_target_group.blue.arn } output "green_tg_arn" { value = aws_lb_target_group.green.arn }
ここまでできたら一旦デプロイします. この時出力される値は後ほど利用するので, 控えておきましょう.
$ terraform apply
VPCやALBの作成が完了した後はTask DefinitionとECS Serviceの作成を行います.
ここだけAWS CLI経由で行っています.
まずはTask Definitionのskeletonを作成します.
YOUR IMAGE URIの部分にはECRリポジトリのURIの記載とexecutionRoleArnを実環境のRole ARNに書き換えてください.
{ "family": "go-server", "networkMode": "awsvpc", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "cpu": "256", "memory": "512", "requiresCompatibilities": ["FARGATE"], "containerDefinitions": [ { "name": "go-server", "image": "YOUR IMAGE URI", "essential": true, "portMappings": [ { "protocol": "tcp", "containerPort": 80, "hostPort": 80 } ] } ] }
次にTask Defnitionを作成します.
$ aws ecs register-task-definition \ --cli-input-json file://task_definition.json \ --region ap-northeast-1
似た手順でECS Serviceも作成します. ARNやIDなど, 指定が必要な部分は先ほどの出力を元に書き換えてください. また今回はパブリックサブネットに配置する都合上, assignPublicIpを有効にしています.
{ "cluster": "mycluster", "serviceName": "go-server", "taskDefinition": "go-server", "loadBalancers": [ { "targetGroupArn": "GREEN TARGET GROUP ARN", "containerName": "go-server", "containerPort": 80 } ], "desiredCount": 1, "capacityProviderStrategy": [ { "capacityProvider": "FARGATE_SPOT", "weight": 1, "base": 1 } ], "networkConfiguration": { "awsvpcConfiguration": { "subnets": ["YOUR SUBNET 1 ID", " YOUR SUBNET 2 ID"], "securityGroups": ["YOUR SERVICE SECURITY GROUP ID"], "assignPublicIp": "ENABLED" } }, "healthCheckGracePeriodSeconds": 0, "deploymentController": { "type": "CODE_DEPLOY" }, "enableECSManagedTags": true }
ECS Serviceを作成します.
$ aws ecs create-service \ --cli-input-json file://service.json \ --region=ap-northeast-1
サービスができた段階で, ALBへのアクセスを行うと正常にレスポンスが返ってくることが確認できます.
$ curl http://xxxxxxxxxxxxxxxx.ap-northeast-1.elb.amazonaws.com/ Hello World
ここまで完了したら, task_definition.jsonを少し書き換えます. 11行目のimageを
{ "family": "go-server", "networkMode": "awsvpc", "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole", "cpu": "256", "memory": "512", "requiresCompatibilities": ["FARGATE"], "containerDefinitions": [ { "name": "go-server", "image": "<IMAGE1_NAME>", "essential": true, "portMappings": [ { "protocol": "tcp", "containerPort": 80, "hostPort": 80 } ] } ] }
次にCodePipelineにDeployステージを追加して, CodeDeploy Applicationを作成します.
既存のコードにDeployステージを追加します.
この時にproviderとして, CodeDeployToECSを利用することとTaskDefinitionTemplatePathとAppSpecTemplatePathのアーティファクトはSourceから, Image1のアーティファクトはBuildから取得するようにします.
Image1は, task_definition.jsonで指定した
resource "aws_codepipeline" "main" { name = "ecs-pipeline" role_arn = aws_iam_role.codepipeline.arn tags = var.tags artifact_store { type = "S3" location = aws_s3_bucket.pipeline_artifact.id } stage { name = "Source" action { name = "Source" category = "Source" owner = "ThirdParty" provider = "GitHub" version = "1" run_order = 1 output_artifacts = ["source"] configuration = { Owner = "YOUR GITHUB ACCOUNT" Repo = "YOUR GITHUB REPOSITORY" Branch = "YOUR GITHUB BRANCH" } } } stage { name = "Build" action { name = "Build" category = "Build" owner = "AWS" provider = "CodeBuild" version = "1" run_order = 2 input_artifacts = ["source"] output_artifacts = ["build"] configuration = { ProjectName = "ecs-pipeline" } } } stage { name = "Deploy" action { name = "Deploy" category = "Deploy" owner = "AWS" provider = "CodeDeployToECS" version = "1" run_order = 3 input_artifacts = ["build", "source"] configuration = { ApplicationName = aws_codedeploy_app.main.name DeploymentGroupName = aws_codedeploy_app.main.name TaskDefinitionTemplateArtifact = "source" TaskDefinitionTemplatePath = "task_definition.json" AppSpecTemplateArtifact = "source" AppSpecTemplatePath = "appspec.yaml" Image1ArtifactName = "build" Image1ContainerName = "IMAGE1_NAME" } } } }
次にCodeDeploy ApplicationとDeployment Groupを作成します. ECSを利用する場合はデプロイ設定についてはAWS側が提供しているものを利用するためそこまで設定も煩雑にはなっていません.
resource "aws_codedeploy_app" "main" { compute_platform = "ECS" name = "go-server" } resource "aws_codedeploy_deployment_group" "main" { deployment_group_name = "go-server" deployment_config_name = "CodeDeployDefault.ECSAllAtOnce" app_name = aws_codedeploy_app.main.name service_role_arn = aws_iam_role.codedeploy.arn auto_rollback_configuration { enabled = true events = [ "DEPLOYMENT_FAILURE" ] } blue_green_deployment_config { deployment_ready_option { action_on_timeout = "CONTINUE_DEPLOYMENT" } terminate_blue_instances_on_deployment_success { action = "TERMINATE" termination_wait_time_in_minutes = 1 } } deployment_style { deployment_option = "WITH_TRAFFIC_CONTROL" deployment_type = "BLUE_GREEN" } ecs_service { cluster_name = aws_ecs_cluster.main.name service_name = "go-server" } load_balancer_info { target_group_pair_info { prod_traffic_route { listener_arns = [ aws_lb_listener.main.arn ] } target_group { name = aws_lb_target_group.blue.name } target_group { name = aws_lb_target_group.green.name } } } }
最後にCodeDeployで利用するIAM Roleを作成します.
iam_role.tf (CodeDeploy)
data "aws_iam_policy_document" "codedeploy_assumerole" { statement { actions = ["sts:AssumeRole"] principals { type = "Service" identifiers = ["codedeploy.amazonaws.com"] } } } resource "aws_iam_role" "codedeploy" { name = "ecs-pipeline-deploy" assume_role_policy = data.aws_iam_policy_document.codedeploy_assumerole.json } resource "aws_iam_role_policy_attachment" "codedeploy" { role = aws_iam_role.codedeploy.id policy_arn = "arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS" }
AWS リソースの作成準備が整ったので, applyします.
$ terraform apply
最後にリポジトリに必要なAppSpecを含めていきます. Type: AWS::ECS::Serviceのインデントが2つなのでそこだけ注意してください.
version: 0.0 Resources: - TargetService: Type: AWS::ECS::Service Properties: TaskDefinition: '<TASK_DEFINITION>' LoadBalancerInfo: ContainerName: 'go-server' ContainerPort: 80
ファイルを記載したら, リポジトリに追加したらCodePipelineの準備が完了です.
リポジトリに変更をpushして動作を確認する
main.go を編集してレスポンス内容を「"Hi, This is New Version!」に変えます.
import ( "fmt" "net/http" ) func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "Hi, This is New Version!\n") }) http.ListenAndServe(":80", nil) }
この変更をリポジトリにpushします.
$ git add main.go $ git commit -m "fix: server response" $ git push -u origin master
CodeDeployのDeploymentを確認してみます.
$ aws deploy list-deployments --region ap-northeast-1 { "deployments": [ "d-XXXXXXXXX" ] } $ aws deploy get-deployment --deployment-id d-XXXXXXXXX --region ap-northeast-1 { "deploymentInfo": { "applicationName": "go-server", "deploymentGroupName": "go-server", "deploymentConfigName": "CodeDeployDefault.ECSAllAtOnce", "deploymentId": "d-XXXXXXXXX", "revision": { "revisionType": "String", "string": { "sha256": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } }, "status": "Succeeded", ~~ }
デプロイの完了を確認したところで実際にALBにアクセスしてみます. 無事にデプロイが完了していることが確認できます.
$ curl http://xxxxxxxxxxxxxxxx.ap-northeast-1.elb.amazonaws.com/ Hi, This is New Version!
さいごに
CodePipelineを1から構築する際には結構はまりどころが多く, それを含めてまとめてみました. 実際に手を動かすことでコンポーネントがどういった動作をするかがつかめたので良かったです. また, CodeDeployのロールバック機能や, CodePipelineの承認や通知, テストステージの準備などまだまだ機能は全て試せてないので今後も試していきたい所存です.